Systems and methods for entering and exiting a teleoperational state

ABSTRACT

A teleoperational system comprises a teleoperational control system and a teleoperational manipulator configured for operating an instrument in an environment. The teleoperational system also comprises an operator controller in communication with the teleoperational control system. The teleoperational control system includes a processing unit including one or more processors. The processing unit is configured to determine whether an operator of the operator controller has a head portion directed toward a display region of a display device and based on a determination that the operator&#39;s head portion is directed toward the display region, initiate an operator following mode in which movement of the operator controller provides a corresponding movement to the teleoperational manipulator. The teleoperational system may be a teleoperational medical system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and benefit of the filing date ofU.S. Provisional Patent Application No. 62/467,506, entitled “Systemsand Methods for Entering and Exiting a Teleoperational State,” filedMar. 6, 2017, which is incorporated by reference herein in its entirety.

FIELD

The present disclosure is directed to systems and methods for ateleoperational procedure and more particularly to systems and methodsfor entering and exiting a teleoperational state.

BACKGROUND

Instruments can be used to manipulate and perform tasks in a work space.Such instruments may be configured to be supported and operatedpartially or entirely by teleoperational manipulator assemblies. Suchinstruments and teleoperational manipulator assemblies can be used toperform non-medical and medical procedures. As a specific example,medical tools or medical manipulators can be used to perform minimallyinvasive medical procedures. As another specific example, industrialtools or industrial manipulators can be used in manufacture or testing.As yet other examples, tools or manipulators can be used in proceduresfor entertainment, exploration, and various other purposes.

As a specific example for minimally invasive medical techniques,minimally invasive medical techniques are generally intended to reducethe amount of tissue that is damaged during invasive medical procedures,thereby reducing patient recovery time, discomfort, and harmful sideeffects. Such minimally invasive techniques may be performed throughnatural orifices in a patient anatomy or through one or more incisions.Through these natural orifices or incisions, clinicians may insertmedical tools to reach a target tissue location. Minimally invasivemedical tools include instruments such as therapeutic instruments,diagnostic instruments, and surgical instruments. Minimally invasivemedical tools may also include imaging instruments such as endoscopicinstruments that provide a user with a field of view within the patientanatomy.

Some medical and non-medical instrument (including manipulationinstruments, imaging instruments or other sensing instruments, etc.) maybe teleoperated or otherwise computer-assisted. When performingteleoperational procedures with remote instrument controllers, safe andreliable mechanisms are desired to determine when the teleoperationalinstruments should be responsive to operator movement of the remoteinstrument.

SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

In one embodiment, a teleoperational medical system comprises ateleoperational control system and a teleoperational manipulatorconfigured for operating a medical instrument in a medical environment.The teleoperational medical system also comprises an operator controllerin communication with the teleoperational control system. Theteleoperational control system includes a processing unit including oneor more processors. The processing unit is configured to determinewhether an operator of the operator controller has a head portiondirected toward a display region of a display device and based on adetermination that the operator's head portion is directed toward thedisplay region, initiate an operator following mode in which movement ofthe operator controller provides a corresponding movement to theteleoperational manipulator.

In another embodiment, a method comprises determining whether a headportion of an operator is directed toward a display region of a displaydevice. Based on a determination that the operator's head portion isdirected toward the display region, an operator following mode isinitiated in which movement of the operator controller provides acorresponding movement to the teleoperational manipulator.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1 provides a view of a surgical environment in which ateleoperational medical system operates, in accordance with anembodiment of the present disclosure.

FIG. 2 illustrates an operator controller according to an embodiment ofthe present disclosure.

FIG. 3 is a flowchart providing a method for entering an activeteleoperational state.

FIGS. 4-6 illustrate optical marker detection systems in accordance withvarious embodiments of the present disclosure.

FIGS. 7 and 8 provide overhead views of a surgical environment in whicha teleoperational medical system operates, in accordance with anembodiment of the present disclosure.

FIG. 9 provides a view of a surgical environment of FIGS. 1, andillustrates the intersection of an operator's view vector with a displayplane.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription of the aspects of the invention, numerous specific detailsare set forth in order to provide a thorough understanding of thedisclosed embodiments. However, it will be obvious to one skilled in theart that the embodiments of this disclosure may be practiced withoutthese specific details. In other instances well known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments of theinvention.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein are for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom).

Although some of the examples described herein refer to surgicalprocedures or tools, or medical procedures and medical tools, thetechniques disclosed apply to medical and non-medical procedures, and tomedical and non-medical tools. For example, the tools, systems, andmethods described herein may be used for non-medical purposes includingindustrial uses, general robotic uses, and sensing or manipulatingnon-tissue work pieces. Other example applications involve cosmeticimprovements, imaging of human or animal anatomy, gathering data fromhuman or animal anatomy, setting up or taking down the system, andtraining medical or non-medical personnel. Additional exampleapplications include use for procedures on tissue removed from human oranimal anatomies (without return to a human or animal anatomy), andperforming procedures on human or animal cadavers. Further, thesetechniques can also be used for medical treatment or diagnosisprocedures that includes, or does not include, surgical aspects.

Referring to FIG. 1 of the drawings, a medical environment 10 thatincludes a teleoperational medical system 12 for use in, for example,medical procedures including diagnostic, therapeutic, or surgicalprocedures. The teleoperational medical system 12 generally includes ateleoperational assembly 13 mounted to or near an operating table O onwhich a patient P is positioned. For simplicity of explanation, portionsof the discussion connected to FIG. 1 will be in connection with asurgical environment implementation of the medical environment 10, andin connection with surgical aspects of the medical procedure. However,the techniques, tools, methods, and apparatuses described in connectionwith FIG. 1 can also be applied to medical procedures outside of asurgical environment and outside of surgery, or to non-medicalprocedures outside of any medical environment.

The teleoperational assembly 13 may include one or more modularmanipulator arms. A medical instrument system 14 or an endoscopicimaging system 15 may be operably coupled to a teleoperationalmanipulator (e.g. an arm) of the teleoperational assembly 13. Anoperator input system 16 allows an operator (e.g. a surgeon, other typeof clinician, other type of medical personnel, or other operator) S tocontrol the operation of the medical instrument system 14 and/or theendoscopic imaging system. One or more other persons (e.g. other medicalpersonnel such as assistant surgeons, anesthesiologists, nurses, medicalassistants other operators, support personnel) A may also be present inthe surgical environment.

For simplicity of explanation, much of this application refers to theoperator S as a surgeon, and the person A as an assistant, within thecontext of a surgical procedure. However, where specialized surgical orassistant skills are not required, the operator S may be a surgeon, someother clinician, some other medical personnel, some non-medicaloperator, or some other person. Similarly, the person A may be anassistant surgeon, some other clinician, some other medical personnel,some non-medical operator, or some other person. Also, where theprocedure performed is not on a patient or is non-medical, (e.g. for anindustrial application, for training, for work on a cadaver or anatomyremoved from and not to be returned to a patient, etc.), the persons Sand A may have little or no medical training or knowledge.

A display system 26 (also called “display 26”) may present imagescaptured by the imaging system 15, surgical navigation and guidanceimages, and/or alphanumeric or symbolic information to assist thesurgeon (or other personnel) S or assistants (or other personnel) A withthe surgical procedure. The teleoperational medical system 12 alsoincludes a control system 28 in communication with the operator inputsystem 16, the teleoperational assembly 13 and the display system 26, asdescribed below.

In this embodiment, the operator input system 16 includes one or a setof operator hand controllers 18 (FIG. 2) as user input devices forcontrolling one or multiple medical instrument systems 14 or theendoscopic imaging system 15. The input system also, optionally,includes a pedal control device 24. The operator hand controllers 18 andthe pedal control may be located at the side of the patient P. Invarious alternatives the operator controllers 18 may be tethered bypower and/or signal transmission cabling or may be untethered/wireless.In other alternative embodiments, the operator controllers may belocated at an operator console such as a surgeon's console, which may belocated in the same room as operating table O. As shown in FIG. 2, theoperator controllers 18 may include one or more of any number of avariety of input devices such as grip inputs 20 and trigger switch 22.The input devices may be used to, for example, close grasping jaw endeffectors, apply an electrical potential to an electrode, deliver amedicinal treatment, or the like. In various alternatives, the operatorinput system may additionally or alternatively include joysticks,trackballs, data gloves, trigger-guns, hand or foot-operatedcontrollers, voice recognition devices, touch screens, body motion orpresence sensors, and the like. In some embodiments, the handcontrollers 18 will be provided with the same degrees of freedom as themedical instruments of the teleoperational assembly to provide thesurgeon (or other personnel) with telepresence, the perception that thecontrol device(s) are integral with the instruments so that the surgeon(or other personnel) S has a strong sense of directly controllinginstruments as if present at the surgical site. In other embodiments,the hand controllers 18 may have more or fewer degrees of freedom thanthe associated medical instruments and still provide the surgeon (orother personnel S with telepresence.

The assembly 13 supports and manipulates the medical instrument system14 while the surgeon (or other personnel) S conducts the procedure fromthe patient side or another location within the surgical environment. Animage of the surgical site within the patient can be obtained by theendoscopic imaging system 15, such as a stereoscopic endoscope, whichcan be manipulated by the teleoperational assembly 13 to orient theendoscopic imaging system 15. The number of medical instrument systems14 used at one time will generally depend on the diagnostic or surgicalprocedure and the space constraints within the operating room amongother factors. Each arm of the teleoperational assembly 13 may include akinematic structure of one or more servo or non-servo controlled links.The teleoperational assembly 13 includes a plurality of motors thatdrive inputs on the medical instrument system 14. These motors move inresponse to commands from the control system 28. The motors includedrive systems which when coupled to the medical instrument system 14 mayadvance the medical instrument into a naturally or surgically createdanatomical orifice. Other motorized drive systems may move the distalend of the medical instrument in multiple degrees of freedom, which mayinclude three degrees of linear motion (e.g., linear motion along the X,Y, Z Cartesian axes) and in three degrees of rotational motion (e.g.,rotation about the X, Y, Z Cartesian axes). Additionally, the motors canbe used to actuate an articulable end effector of the instrument forgrasping tissue in the jaws of a biopsy device or the like. Instruments14 may include end effectors having a single working member such as ascalpel, a blunt blade, an optical fiber, or an electrode. Other endeffectors may include, for example, forceps, graspers, scissors, or clipappliers.

The control system 28 includes at least one memory and at least oneprocessor, and typically a plurality of processors, for effectingcontrol between the medical instrument system 14, the imaging system 15,the operator input system 16, the display system 26, and other auxiliarysystems which may include, for example, additional imaging systems,audio systems, fluid delivery systems, display systems, illuminationsystems, steering control systems, irrigation systems, and/or suctionsystems. The control system 28 also includes programmed instructions(e.g., a computer-readable medium storing the instructions) to implementsome or all of the methods described in accordance with aspectsdisclosed herein. While control system 28 is shown as a single block inthe simplified schematic of FIG. 1, the system may include two or moredata processing circuits with one portion of the processing optionallybeing performed on or adjacent the teleoperational assembly 13, anotherportion of the processing being performed at the operator input system16, and the like. In various embodiments, the control system may behoused in an electronics cart 30 to which the display system 26 or otherperipheral equipment is mounted. The control system 28 may employ any ofa wide variety of centralized or distributed data processingarchitectures. Similarly, the programmed instructions may be implementedas a number of separate programs or subroutines, or they may beintegrated into a number of other aspects of the teleoperational systemsdescribed herein. In one embodiment, control system 28 supports wirelesscommunication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11,DECT, and Wireless Telemetry.

In some embodiments, control system 28 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system 14. Responsive to the feedback, the servo controllerstransmit signals to the operator input system 16. The servocontroller(s) may also transmit signals instructing teleoperationalassembly 13 to move the medical instrument system(s) 14 and/ orendoscopic imaging system 15 which extend into an internal surgical sitewithin the patient body via openings in the body. Any suitableconventional or specialized servo controller may be used. A servocontroller may be separate from, or integrated with, teleoperationalassembly 13. In some embodiments, the servo controller andteleoperational assembly are provided as part of a teleoperational armpositioned adjacent to the patient's body.

The control system 28 can be coupled with the imaging system 15 and caninclude a processor to process captured images for subsequent display,such as to a surgeon (or other personnel) on a console such as asurgeon's console, or on another suitable display located locally and/orremotely. For example, where a stereoscopic endoscope is used, thecontrol system 28 can process the captured images to present the surgeon(or other personnel) with coordinated stereo images of the surgicalsite. Such coordination can include alignment between the opposingimages and can include adjusting the stereo working distance of thestereoscopic endoscope.

In alternative embodiments, the teleoperational system may include morethan one teleoperational assembly and/or more than one operator inputsystem. The exact number of teleoperational assemblies will depend onthe surgical procedure and the space constraints within the operatingroom, among other factors. The operator input systems may be collocatedor they may be positioned in separate locations. Multiple operator inputsystems allow more than one operator to control one or more manipulatorassemblies in various combinations.

During a minimally invasive teleoperational procedure, the surgeon (orother personnel) S may view the surgical site within the patient's bodyon the display system 26 via images captured by the imaging system 15.The “following mode” of the teleoperational medical system 12 is asystem operational state in which the movement of the controllers 18effect movement of the instruments 14. Generally, when operating thecontrollers 18 in following mode, the surgeon (or other personnel) Sviews the images on the display system 26 to carefully control theinstruments 14 contacting the patient P. To safely perform ateleoperational procedure, movement of the instruments 14 within thepatient anatomy may be suspended when the surgeon (or other personnel) Slooks away from the display system or moves to a position inside oroutside of the surgical environment in which the display system is notvisible. In teleoperational systems where the surgeon operates from aconsole such as a surgeon's console with a head-in display system,sensors or switches can determine when the surgeon's face is moved awayfrom contact with the console. In patient-side teleoperational systems,the surgeon's head may not be positioned within a console and thereforedetecting whether the surgeon is viewing the display system from adistance (e.g. across the operating table 0 or across the room) may bedetermined by other means. Safety systems described below may be used todetermine whether the surgeon (or other personnel) S is viewing theendoscopic images of the surgical site on the display system and mayenable or continue an operator following mode when one or more viewingcriteria is met.

FIG. 1 illustrates a spatial tracking system 32 that detects objects ormarkers within a spatial volume 34 of the surgical environment 10. Inthis embodiment, the spatial tracking system 32 may be an opticaltracking system, but in various alternative embodiments, other spatialtracking systems such as acoustic, eye or gaze tracking,electromagnetic, inertial measurement unit (IMU), or hybrid trackingsystems may be used. In some embodiments, an optical tracking system mayhave multiple uses beyond tracking the surgeon's view. For example, thetracking system may also be used for tracking hand motion or for imageregistration.

In the embodiment of FIG. 1, the tracking system 32 is generally fixedrelative to the coordinate frame of the display system 26. For example,the tracking system 32 may be mounted to the display system 26 to tracka volume 34 in front of the display. The tracking system 32 may trackmarkers within the volume 34 that indicate whether the surgeon (or otherpersonnel) S, who is spaced a distance apart from the display system 26,is looking at a display region of the display system. In alternativeembodiments, the tracking system may be mounted elsewhere in thesurgical environment 10 and may use a transformation between the displaysystem position coordinates and the tracking system coordinates todetermine the position and orientation of markers or objects relative tothe display system. In alternative embodiments, the tracking system mayinclude a camera or other sensor coupled to the surgeon's head whichtracks the motion of the surgeon's head and/or eyes relative to staticor movable markers with known positions in the surgical environment. Inalternative embodiments, the display system may be a virtual displaysystem that has a fixed or known virtual projection plane within thesurgical environment. In alternative embodiments, the display system maybe a virtual display system projected in a head-mounted augmentedreality headset worn by the surgeon (or other personnel) S. The movementof the surgeon's (or other personnel's) head and/or eyes may be trackedusing, for example, simultaneous localization and mapping (SLAM)techniques.

FIG. 3 illustrates a method 100 for entering an active teleoperationalstate such as an operator following mode of the teleoperational system12. The method 100 is illustrated in FIG. 3 as a set of operations orprocesses 102-114. Not all of the illustrated processes 102-114 may beperformed in all embodiments of method 100. Additionally, one or moreprocesses that are not expressly illustrated in FIG. 3 may be includedbefore, after, in between, or as part of the processes 102-110. In someembodiments, one or more of the processes may be implemented, at leastin part, in the form of executable code stored on non-transitory,tangible, machine-readable media that when run by one or more processors(e.g., the processors of control system) may cause the one or moreprocessors to perform one or more of the processes.

According to the present example, the method includes a process 102 fordetermining whether an operator's (e.g. surgeon's or other personnel's)view is directed toward a display region of a display system (e.g.display system 26). This determination may be made, for example, usingthe tracking system 32. The tracking system 32 may detect markers,including natural landmarks on a portion of the operator's head.Examples of markers tracked by the tracking system 32 are illustrated inFIGS. 4-6.

FIG. 4 illustrates an operator 150 wearing a marker 152 that may betracked by the tracking system 32. In this embodiment, the marker 152 isan optical marker that is rigidly coupled to stereo viewing glasses 154worn by the operator 150. The glasses 154 hold the marker 152 in agenerally fixed position relative to the operator's eyes. The positionof the operator's eyes relative to the glasses and marker may bemeasured or known. Based on the known position of the marker 152relative to the glasses and the known position of the operator's eyesrelative to the glasses, the position of the eyes relative to thetracking system 32 may be determined by the tracking system. In thisembodiment, the marker is coupled to stereo viewing glasses which theuser wears to obtain a 3D view of the images displayed on the displaysystem 26. In alternative embodiments, for example, when glasses are nototherwise used by the operator, the marker may be attached to anotheraccessory coupled to the operator's head such as a head band or anearpiece that may be held generally fixed relative to the operator'seyes. In alternative embodiments, the marker may be attached directly tothe operator's head with an adhesive or other coupling device.

FIG. 5 illustrates the operator 150 wearing a set of one or more opticalmarkers 156 that may be tracked by the tracking system 32. In thisembodiment, the markers 156 are rigidly attached to glasses 154 worn bythe operator 150. The glasses 154 hold the markers 156 in a generallyfixed position relative to the operator's eyes. The position of theoperator's eyes relative to the glasses and marker may be measured orknown. In this embodiment, the markers 156 frame the operator's eyes,providing a constellation of trackable points that have a generallyfixed position relative to the operator's eyes, which allows theposition of the eyes to be calculated by the tracking system 32. Invarious embodiments, ordinary features of the glasses such as corners orthe bridge may serve as trackable markers.

FIG. 6 illustrates natural markers 158-164 corresponding to naturalportions of the head of operator 150. These natural markers 158-164 maybe tracked by the tracking system 32. For example, natural markers 158may be the pupils or glint points of the operator's eyes. The naturalmarkers 158 may be used to track the operator's gaze. Natural markers160 may mark the outer edges of the operator's nose. Natural markers 162may mark the outer edges of the operator's mouth. Natural marker 164 maycorrespond to the lowest point of the operator's chin. The naturalmarkers 160-164 may have known positions relative to the operator'seyes. Other markers such as the outline of the operator's face or aderived midline defined between the markers 160, 162, and 164 may alsobe tracked by the tracking system.

Referring again to the method 100 of FIG. 3, at process 102 tracking ofone or more markers (including natural landmarks or artificial markers)may be used to estimate the position and orientation of the operator'shead, face, eyes, gaze, or other head portion. The positions andorientations may be used to compute the looking direction as a viewingvector in the coordinate frame of the display system. FIG. 7 illustratesa viewing vector 170 computed using tracked markers on the surgeon (orother personnel) S. The viewing vector may be, for example, based uponhead orientation, eye gaze direction or a combination of the two.

FIG. 9 illustrates a display plane 172 which is coincident with thesurface of the display system 26. A display region 174 may be smallerthan the full display field 176 of the display system 26 viewable by thesurgeon (or other personnel) S. The display region 174 may includecurrent endoscopic images or other information critical to the medicalprocedure. The display plane 172 may include multiple display regionscontaining different images or information.

At the process 102, to determine whether the operator's view is directedtoward the display region 174, an intersection point 178 between theviewing vector 170 and the display plane 172 may be computed. Based onthe location of the intersection point 178, the determination can bemade as to whether the intersection point 178 is within the boundariesof the display region 174. If the operator's view is within the displayregion 174, then the operator following mode may be initiated at process110, allowing the operator's movement of the controllers 18 to translateinto movement of the instrument 14. If the operator's view is notdirected toward the display region, the entry into following mode isprevented. This serves as a safety mechanism to prevent movement of theinstruments 14 within the patient anatomy when the operator looks awayfrom the display region. In alternative embodiments, the determinationof the operator's view may be made by computing whether a view angle 180(FIG. 7) is within permitted bounds of the display region 174 or thedisplay field 176.

Optionally, transitioning to the operator following mode may becontingent on the evaluation of additional conditions. For example, atoptional process 106, the tracking system may determine whether theoperator's head is within an operating volume 182 (FIG. 8) located in apredefined position relative to the display system 26. The operatingvolume 182 may define an acceptable position within the surgicalenvironment 10 from which the procedure may be performed. If theoperator is not located within the operating volume, entry into thefollowing mode may be prevented. This also serves as a safety mechanismto prevent movement of the instruments 14 within the patient anatomywhen the operator walks away from the accepted location for performingthe procedure.

Optionally, transitioning to the operator following mode may becontingent on the evaluation of an activation gesture. For example, atoptional process 108, the control system 28 may evaluate whether thesurgeon (or other personnel) S has performed an activation gesture. Theactivation gesture may be, for example, the operator forming a matchinggrip in which the controller 18 is moved to match the configuration ofthe instrument 14 visible in the endoscopic image on the display system26. Matching the configuration may include orienting the controller 18at the same angle as the instrument and opening or closing the fingersin the grip inputs 20 of the controller to match the jaw spacing of ajawed instrument 14 visible in the display region 174. Other activationgestures may include a hand gesture sensed by the tracking system 32, apedal movement, button activation, a head gesture, or a verbal command.

Optionally, a variety of other conditions may be required for enteringan operator following mode. For example, operator recognition may berequired before entering the operator following mode. Operatorrecognition may be determined using, for example, a facial recognitionsystem, an iris recognition system, a fingerprint recognition system,another biometric identification system, or a voice command.Additionally or alternatively the operator may be required to inputidentification information or may wear a personal identification markerthat may be sensed before entering the following mode. Additionally oralternatively, a presence detector may be used to determine theoperator's presence at the controllers 18. The presence detector may be,for example, a capacitive touch sensor.

Another optional criterion before entering the operator following modeat process 110, may be a threshold viewing time period. For example, theoperator may be required to maintain the viewing vector or viewing anglefor the threshold time period as an indication of intention to performthe procedure and not as an accidental glance.

Referring again to FIG. 3, at a process 112, the surgeon's (or otherpersonnel's) direction of view continues to be monitored by the trackingsystem 32 throughout the procedure to determine whether the operator'sdirection of view moves away from the display region 174. As previouslydescribed, the operator's direction of view may be determined based on acalculated viewing vector or angle. The optional criteria of operator'shead position may also be monitored during the surgical procedure todetermine whether the operator moves out of the permitted spatialregion. Other optional criteria for maintaining the operator followingmode may also be monitored throughout the surgical procedure.

At a process 114, the teleoperational system may exit or suspend theoperator following mode if the operator's direction of view moves awayfrom the display region or if any of the optional criteria beingmonitored are no longer satisfied. The exiting from the following modemay occur immediately upon determination that a criterion is no longerbeing met. Alternatively, the exiting from the following mode may occurafter criterion fails to be met for a predefined threshold period oftime. For example, if the operator glances away from the display regionfor less than the predefined threshold period of time (e.g., 500milliseconds), the system may remain in following mode. As anotherexample, if the operator glances away from the display region for lessthan the predefined threshold period of time (e.g., 500 milliseconds),the system may enter a temporary clutched state in which movement of thecontrollers 18 does not cause movement of the instruments in the patientanatomy while the operator is looking away. That is, if the operator'sdirection of view moves away and then returns to the display regionbefore the threshold period of time elapses, the system enters into thetemporary clutched state when the operator's direction of view has movedaway, and then the system exists the temporary clutched state andresumes full following mode when the operator's direction of view hasreturned. But, if the operator looks away from the display region for atime period that exceeds the predefined threshold time period, thesystem may terminate following mode so that movement of the controllers18 no longer causes movement of the instruments in the patient anatomywhile the operator is looking away. When following mode is terminated,the system may transition into a standby state or to a master clutchstate. To return to the following mode after termination, the operatormay reengage with an activation gesture.

To enter a master clutch state, the user begins in following mode. Withuser inputs such gestures or hand or foot pedal activation, the user mayenter into clutch mode. In clutch mode, the teleoperational instrumentor endoscope doesn't move while the teleoperational controllers aremoving. When teleoperational controllers are moved, the clutch statewill allow the user to reset hand position or pause briefly for anyreason while still looking at surgical video. The clutch mode may putthe system back into following mode without performing an activationgesture. Standby mode is similar to a clutch mode, but the user mayenter a standby mode without performing active gestures. Instead a usermay enter standby mode if he or she is not looking at an active regionof the display for more than a predefined time. When in standby mode,the user may return to following mode by looking into the active regionor may exit following mode when not looking at the active region formore than a predefined time. When the system enters standby move, theteleoperational instruments do not follow the movement of theteleoperational controllers, as in clutch mode.

Optionally, the display may include multiple display regions, with eachdisplay region having a different predefined time period for determiningsuspension of the following mode. In various embodiments, one or more“safe” regions may be established in or out of the display plane suchthat when the view direction moves to the safe regions a different,longer predefined time period governs suspension of the following mode.Such safe regions may be established, for example, in the viewingdirection of the patient anatomy so that the operator may look down atthe patient surface for a time period greater than the predefined timeperiod, without the teleoperational system exiting the following mode.Optionally, exiting of the following mode may be accompanied by a visual(e.g. flashing light) or audio (e.g. an audible tone) indicator.

Alternatively, the exiting from the following mode may occur based on adistance between display region and the direction of the operator'sview. For example, the time to exit the following mode may be shorter ifthe operator's view vector intersects the display plane far away fromthe display region (indicating the operator has looked far away from thedisplay region), and the transition out of following mode may occurafter a longer period of time if the view vector intersects the displayplane at a location just barely outside of the display region(indicating that the operator may just be looking at other relevantinformation on the display system.

One or more elements in embodiments of the invention may be implementedin software to execute on a processor of a computer system such ascontrol processing system. When implemented in software, the elements ofthe embodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable storage medium or device that may have beendownloaded by way of a computer data signal embodied in a carrier waveover a transmission medium or a communication link. The processorreadable storage device may include any medium that can storeinformation including an optical medium, semiconductor medium, andmagnetic medium. Processor readable storage device examples include anelectronic circuit; a semiconductor device, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable read only memory (EPROM); a floppy diskette, a CD-ROM, anoptical disk, a hard disk, or other storage device, The code segmentsmay be downloaded via computer networks such as the Internet, Intranet,etc.

Note that the processes and displays presented may not inherently berelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct a morespecialized apparatus to perform the operations described. The requiredstructure for a variety of these systems will appear as elements in theclaims. In addition, the embodiments of the invention are not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart.

1. A teleoperational medical system comprising: a teleoperationalcontrol system; a teleoperational manipulator configured for operating amedical instrument in a medical environment; and an operator controllerin communication with the teleoperational control system, wherein theteleoperational control system includes a processing unit including oneor more processors, and wherein the processing unit is configured to:determine whether a head portion of an operator of the operatorcontroller is directed toward a display region of a display device basedon a position and an orientation of a marker on the operator, whereinthe position and the orientation of the marker relative to the headportion of the operator remain substantially fixed during movement ofthe head portion, and based on a determination that the head portion isdirected toward the display region, initiate an operator following modein which movement of the operator controller provides a correspondingmovement to the teleoperational manipulator.
 2. The teleoperationalmedical system of claim 1 wherein the processing unit is configured toprevent initiation of the operator following mode responsive to adetermination that the head portion is not directed toward the displayregion.
 3. The teleoperational medical system of claim 1 wherein theprocessing unit is configured to prevent initiation of the operatorfollowing mode responsive to a determination that the operator is notpresent at the operator controller. cm 4-5. (canceled)
 6. Theteleoperational medical system of claim 1, wherein the marker includes anatural landmark on a face of the operator. 7-9. (canceled)
 10. Theteleoperational medical system of claim 1, wherein the processing unitis further configured to determine whether the head portion ispositioned in a predefined region in the medical environment and whereininitiating the operator following mode is further based on adetermination that the head portion is positioned in the predefinedregion.
 11. The teleoperational medical system of claim 1, wherein theprocessing unit is further configured to determine whether the operatorhas performed an activation gesture and wherein initiating the operatorfollowing mode is further based on a determination that the operator hasperformed the activation gesture.
 12. The teleoperational medical systemof claim 11 wherein the activation gesture is a matching hand grip thatmatches a displayed configuration of the medical instrument on thedisplay device or a movement of a pedal device.
 13. (canceled)
 14. Theteleoperational medical system of claim 1, 3, wherein the processingunit is further configured to identify the operator, and whereininitiating the operator following mode is further based on adetermination that the identified operator is permitted to operate theteleoperational medical system.
 15. (canceled)
 16. The teleoperationalmedical system of claim 1, wherein the processing unit is furtherconfigured to determine whether the head portion is directed away fromthe display region of the display device, and based on a determinationthat the head portion is directed away from the display region, suspendthe operator following mode such that movement of the operatorcontroller does not provide the corresponding movement to theteleoperational manipulator.
 17. (canceled)
 18. The teleoperationalmedical system of claim 16, wherein determining whether the head portionof the operator is directed away from the display region of a displaydevice includes determining whether the head portion of the operator isdirected away from the display region for at least a threshold timeperiod, wherein the threshold time period is a first threshold timeperiod for a first direction that the head portion is directed away fromthe display region, wherein the threshold time period is a secondthreshold time period for a second direction that the head portion isdirected away from the display region, wherein the first threshold timeperiod differs from the second threshold time period, and wherein thefirst direction differs from the second direction.
 19. Theteleoperational medical system of claim 1, wherein determining whetherthe operator of the operator controller has a head portion directedtoward the display region of the display device includes determining anoperator viewing vector and determining whether the operator viewingvector intersects a display plane within the display region. 20.(canceled)
 21. A method for operating a teleoperational system includinga teleoperational manipulator and an operator controller, the methodcomprising: determining whether a head portion of an operator isdirected toward a display region of a display device based on a positionand an orientation of a marker on the operator, wherein the position andthe orientation of the marker relative to the head portion of theoperator remain substantially fixed during movement of the head portion;and based on a determination that the head portion is directed towardthe display region, initiating an operator following mode in whichmovement of the operator controller provides a corresponding movement tothe teleoperational manipulator.
 22. The method of claim 21 furthercomprising: preventing initiation of the operator following moderesponsive to a determination that the head portion is not directedtoward the display region or that the operator is not present at theoperator controller. 23-26. (canceled)
 27. The method of claim 21,further comprising determining whether the head portion is positioned ina predefined region and wherein initiating the operator following modeis further based on a determination that the head portion is positionedin the predefined region.
 28. The method of claim 21, further comprisingdetermining whether the operator has performed an activation gesture andwherein initiating the operator following mode is further based on adetermination that the operator has performed the activation gesture.29-31. (canceled)
 32. The method of claim 21, further comprising:determining whether the head portion is directed away from the displayregion of the display device; and based on a determination that the headportion is directed away from the display region, suspending theoperator following mode such that movement of the operator controllerprovides no corresponding movement to the teleoperational manipulator.33-34. (canceled)
 35. The method of claim 32 further comprising:determining that the head portion has been directed away from thedisplay region of the display device for less than a threshold timeperiod and has returned to be directed toward the display region; andreturning to the operator following mode when the head portion returnsto be directed toward the display region. 36-37. (canceled)
 38. Theteleoperational medical system of claim 1, wherein the marker comprises,or is coupled, to an item worn by the operator, the item selected fromthe group consisting of: glasses, a headband, and an earpiece.
 39. Theteleoperational medical system of claim 16, wherein the processing unitis further configured to: determine that the head portion has beendirected away from the display region of the display device for lessthan a threshold time period and has returned to be directed toward thedisplay region; and return to the operator following mode when the headportion returns to be directed toward the display region.
 40. Anon-transitory machine-readable medium comprising a plurality ofmachine-readable instructions which when executed by one or moreprocessors associated with a medical device are adapted to cause the oneor more processors to perform a method comprising: determining whether ahead portion of an operator is directed toward a display region of adisplay device based on a position and an orientation of a marker on theoperator, wherein the position and the orientation of the markerrelative to the head portion of the operator remain substantially fixedduring movement of the head portion; and based on a determination thatthe head portion is directed toward the display region, initiating anoperator following mode in which movement of an operator controllerprovides a corresponding movement to a teleoperational manipulator.